This video examines the feature of Watson studio that helps to ensure fairness and explain-ability of machine learning pipelines, as well as monitored their performance after deployment.
IBM Watson Openscale is a product that includes several important features. It can test the model
and its predictions for fairness and apply ways to overcome bias. It can also help to
provide explanations for model predictions that are often hard to get but are necessary
for compliance in some application areas. It monitors the model performance and can detect
its deterioration or model drift over time. It can alert the users when drift is detected and
explain which predictors are causing it. We can specify criteria under which the model
gets automatically retrained on fresh data; it also helps to measure how the model helps
the business. The attributes to monitor for bias are automatically recommended based on
prior experience. They can be edited as needed. Openscale then keeps track of model predictions
for the specified groups and checks the bias in the predictions. Users need to know that
their AI models are fair but the date of their models were trained on and include
unwanted biases\a which may unintentionally be included in the resulting models. IBM Watson
Openscale can detect bias when a model is in production and not just when it's being built.
In this demo of Watson Openscale we'll monitor a credit risk model which has been trained to
determine whether or not someone is eligible for a loan, based on a variety of different features,
such as their credit history age and their number of dependents. After launching Openscale we can
see a few highlighted metrics for the monitored model, such as its quality and a fairness score.
What Openscale does is measure a model's fairness by calculating the difference between the rates at
which different groups, for example, women versus men, received the same outcome. A fairness value
below 100% means that the monitored group receives an unfavorable outcome more often
than the reference group. In this case, we see that women are receiving the no-risk outcome,
or getting approved for loans, at a lower rate than men. Openscale
enables the inspection of each model's training data and this reveals that there was more training
data for men than women. This can give some insight as to why the model exhibits bias against
women who apply for loans. Data scientists can use this information to approve the model. Now,
detecting bias is one thing-- Openscale can also mitigate it by creating a D bias model that runs
alongside the monitored one. In this case the D bias model is 12% more fair than the production
model. The D bias model has been trained to detect when your production model will make
a bias prediction so that you can isolate the specific transactions that result in the bias.
For each of these transactions Watson Openscale will flip the monitored value in a record to the
reference value, in this case from female to male, and leave all other data points in that
record the same. If this changes the prediction from risk to no-risk then the D biased model
will surface the no-risk outcome as the D biased result. This is just one of the ways that Watson
open scale helps you ensure that your models are fair explainable and compliant wherever your model
was built or is running. Insurance underwriters can use machine learning and Openscale to more
consistently and accurately assess claims risk, ensure fair outcomes for customers, and explain
AI recommendations for regulatory and business intelligence purposes. Why does an AI model arrive
at a given recommendation or prediction? Users and customers want an explanation and with most
models providing this information is not an easy task. IBM Watson Openscale explains predictions
in business friendly language. This credit application, for instance, was predicted to
be a risk. Openscale determines the features which contributed positively or negatively
to that prediction and spells them out. The explanation is presented visually, as well as in
a sentence-based text summary in order to ensure maximum clarity. Using proprietary IBM research
technology, Openscale also generates a contrast of explanations. Here we see the minimum changes to
this input record which would produce a different output, changing the prediction from risk to
no-risk. The explanations provided by Watson Openscale can help organizations comply with
regulations such as the Fair Credit Reporting Act and GDPR which give customers the right to ask for
reasons why their applications were denied. Before an AI model is put into production it must prove
it can make accurate predictions on test data, a subset of its training data; however, over time,
production data can begin to look different than training data, causing the model to start making
less accurate predictions. This is called drift. IBM Watson Openscale monitors a model's accuracy
on production data and compares it to accuracy on its training data. When a difference in accuracy
exceeds a chosen threshold Openscale generates an alert. Watson Openscale reveals which transactions
caused drift and identifies the top transaction features responsible. For instance, 25% of a
transactions causing drift in this loan approval model were problematic because of these features,
which contained data crucially different from the training data. The transactions causing drift can
be sent for manual labeling and use to retrain the model so that its predictive accuracy does not
drop at run time. Watson Openscale not only helps identify drift but also highlights its root cause
and provides transactions which can be turned into training data useful at fixing drift. It gives you
the insight you need to ensure that your models will consistently deliver the results you want
over time. For instance, the retrain version of the model, but based on the recommendations made
by Watson Openscale, started making accurate recommendations alleviating the drift. This is
just one of the ways that Watson Openscale helps you ensure your models are fair explainable and
compliant wherever your model was built or is running. In this video you have learned how
Openscale ensures fairness and explain ability of models and monitors for model drift in production.
This completes the model on IBM products for data scientists. Good luck on the quizzes!